Healthcare Fraud Detection¶
Author: Huizhe (Sunny) Zhu- hz2657¶
Medical charges vary widely by region and type of work and other factors. This project aims to claim a doctor who charges high a fraud. We need to conclude what is a fair comparison, by considering the type of medical work (DRG) and state,
First, we construct features/variables to identify fraud. Then, we use Kmeans clustering method to identify outliers.
Dataset description: https://data.cms.gov/Medicare-Inpatient/National-Summary-of-Inpatient-Charge-Data-by-Medic/efwk-h4x3
0. load packages, import dataset¶
In [1]:
import numpy as np
import pandas as pd
In [2]:
df = pd.read_csv('inpatientCharges.csv')
Variable description
DRG Definition: Classification system that groups similar clinical conditions (diagnoses) and the procedures furnished by the hospital during the stay.
Total discharges: The number of discharges billed by the provider for inpatient hospital services. When you leave a hospital after treatment, you go through a process called hospital discharge.
Average Covered Charges = Total Covered Charge Amount / Total Discharges, Total Covered Charge Amount = the sum of all covered charges, Covered Charges: Charges for covered services that your health plan paid for.
Average Total Payments = Total Payments / Total Discharges, Payment is the amount a hospital actually receives for providing patient care. This is the actual amount paid to a hospital by consumers, insurers or governments.
Average Medicare Payments = Medicare Payment Amount / Total Discharges, Medicare Payment Amount: The average amount that Medicare pays to the provider for Medicare's share of the MS-DRG.
In [3]:
# put rows have the same provider id together, sort values for provider id
df = df.sort_values('Provider Id', ascending = True)
df.head(3)
Out[3]:
1. EDA¶
- The dataset contains 163065 columns, 12 rows, there is no na value
- The column contains the types of clinical conditions, provider details (id, name, address, city, state, zip code), the number of discharge for a clinical condition, average coverage charged billed to insurance company, the average of total payments received from insurance company, and the average of medicare payment.
- The last three columns are 'object', we convert last three columns to float and remove the whitespace in the column names
In [4]:
# 163065 columns, 12 rows
df.shape
Out[4]:
In [5]:
# examine column names
df.columns
Out[5]:
In [6]:
# check data types
df.dtypes
Out[6]:
In [7]:
# convert last three 'object' columns to float
df[df.columns[9:]] = df[df.columns[9:]].replace('[\$,]', '', regex=True).astype(float)
# rename column names
df.columns = ['DRG Definition', 'Provider Id', 'Provider Name',
'Provider Street Address', 'Provider City', 'Provider State',
'Provider Zip Code', 'Hospital Referral Region Description',
'Total Discharges', 'Average Covered Charges',
'Average Total Payments', 'Average Medicare Payments']
No missing value in this dataset
In [8]:
# by using .info(), we can see there is no na value in the dataset
df.info()
# alternatively, we can use df.isnull() to check missing values
df.isnull().sum()
Out[8]:
In [9]:
# Here we can see the average and standard devition for average covered charged, average total payments, and average medicare payment
# we can see average covered charged is more than average total payments, which is more than average medicare payments
df.describe()
Out[9]:
Exploratory graph¶
In [10]:
# graph 1: distribution of total discharges
import matplotlib.pyplot as plt
plt.hist(df['Total Discharges'], 30, range=[0, 500], align='mid')
plt.ylabel('total discharges', fontsize=14)
plt.title('total discharges distribution', fontsize=14)
Out[10]:
In [11]:
# graph 2: distribution of Average Covered Charges
plt.hist(df['Average Covered Charges'], 30, range=[0, 400000], align='mid')
plt.ylabel('Average Covered Charges', fontsize=14)
plt.title('Average Covered Charges', fontsize=14)
Out[11]:
we can see from the graph that average total payments tend to be 10 times less than average covered charges
In [12]:
# graph 3: distribution of Average Total Payments
plt.hist(df['Average Total Payments'], 30, range=[0, 40000], align='mid')
plt.ylabel('Average Total Payments', fontsize=14)
plt.title('Average Total Payments', fontsize=14)
Out[12]:
we can see from the graph that the distribution of average Medicare payments is very similar to average total payments
In [13]:
# graph 4: distribution of Average Medicare Payments
plt.hist(df['Average Medicare Payments'], 30, range=[0, 40000], align='mid')
plt.ylabel('Average Medicare Payments', fontsize=14)
plt.title('Average Medicare Payments', fontsize=14)
Out[13]:
Average covered charged by state
- California has the highest 'Average Covered Charges'
In [14]:
# graph 5: Average Covered Charges by State
import seaborn as sns
import matplotlib.pyplot as plt
a4_dims = (15.7, 8.27) # default figure size
fig, ax = plt.subplots(figsize=a4_dims)
sns.set(style="whitegrid")
sns.boxenplot(x="Provider State", y="Average Covered Charges",
color="b", scale="linear", data=df,ax=ax)
ax.set_title('Average Covered Payment Hospital Received by State')
Out[14]:
Average total payment recieved by hospitals by state
- New York and California has the highest 'Average Total Payment'
In [15]:
# graph 6: Average total payment recieved by hospitals by state
a4_dims = (15.7, 8.27) # default figure size
fig, ax = plt.subplots(figsize=a4_dims)
sns.set(style="whitegrid")
sns.boxenplot(x="Provider State",
y="Average Total Payments",
color="b", scale="linear", data=df,ax=ax)
ax.set_title('Average Total Payment Hospitals received by State')
Out[15]:
Average total payment recieved by hospitals by state
- New York, California, and Texas have the highest 'Average Medicare Payments' received from government.
In [16]:
# graph 7: Average total payment recieved by hospitals by state
a4_dims = (15.7, 8.27) # default figure size
fig, ax = plt.subplots(figsize=a4_dims)
sns.set(style="whitegrid")
sns.boxenplot(x="Provider State",
y="Average Medicare Payments",
color="b", scale="linear", data=df, ax=ax)
ax.set_title('Average Medicare Payments Hospitals received by State')
Out[16]:
2. construct features¶
we use aggregation strategy to generate features:
we use median, given the dataset has extreme values
--- discharge by provider
- total number of discharge by provider
- group by 'Provider Id', sum('Total Discharges')
--- ratio by clinical condition ----
- ratio of average coverage by a provider / median coverage for this clinical condition in the state
- group by 'Provider State', median('Average Covered Charges')
- ratio of average total payment to a provider/ median payment for this clinical condition in the state
- group by 'Provider State', median('Average Covered Charges')
- ratio of average total medicard coverage paid to a provider/ median medicare coverage for this clinical condition in the state
- group by 'Provider State', median('Average Covered Charges')
--- ratio of discharge by city ----
- ratio of total number of discharge by provider to the median level of discharges in the same city
Feature 1. total number of discharge by provider¶
- group by 'Provider Id', sum('Total Discharges')
Why the feature can identify fraud: we can examine by selected city, which hospital has extreme number of total discharges. We can further examine the outliers.
In [17]:
# create variable: discharge_by_provider
discharge_by_provider = df.groupby(['Provider Name', 'Provider Id', 'Provider City'])['Total Discharges'].sum()
discharge_by_provider = pd.DataFrame(discharge_by_provider)
discharge_by_provider.columns = ['discharge_by_provider']
df2 = pd.merge(df, discharge_by_provider, how='left', on=['Provider Name', 'Provider Id', 'Provider City'])
df2.head(2)
Out[17]:
In [18]:
# bring index to columns
discharge_by_provider = discharge_by_provider.reset_index()
# select city: NEW YORK
selected = discharge_by_provider.loc[discharge_by_provider['Provider City'] == 'NEW YORK']
# here we can see the total number of discharge by provider in New York
selected.head(2)
Out[18]:
There are 11 providers in New York, from the graph below we can see the number of discharges for each provider in NYC.
From the graph we can see, New York Presbyterian hospital has the most number of discharges.
In [19]:
# data visualization
# select city = 'New York'
# x: Provider Name
# y: discharge_by_provider
import plotly
import plotly.express as px
fig = px.scatter(selected, x="Provider Name", y="discharge_by_provider", color = 'discharge_by_provider', size = 'discharge_by_provider',
title = "Total number of discharge for providers in New York",
size_max=60,width=800, height=500)
fig.show()
#sns.set(style="whitegrid")
#sns.boxplot(x="Provider Name", y="discharge_by_provider",
# data=selected_df2)
In [20]:
# examine which city is included in the dataset
#for i in df2['Provider City'].unique():
# print(i)
----- the following are improved/revised features -----¶
Feature 2. ratio of average coverage by a provider / median coverage for this clinical condition in the state¶
- group by 'DRG Definition', 'Provider State', median('Average Covered Charges')
Why the feature can identify fraud: we can examine by selected city, the ratio of average coverage for one clinical condition to the average of coverage for that clinical condition in the state
In [21]:
# create variable: average_coverage_by_DRG_by_state
median_coverage_by_DRG_by_state = df2.groupby(['DRG Definition', 'Provider State'])['Average Covered Charges'].median()
# create average spending group by agency and merchant category
median_coverage_by_DRG_by_state = pd.DataFrame(median_coverage_by_DRG_by_state)
median_coverage_by_DRG_by_state.columns = ['median_coverage_by_DRG_by_state']
# add the new feature average spending to the dataset
df2 = pd.merge(df2, median_coverage_by_DRG_by_state, how='left', on=['DRG Definition', 'Provider State'])
df2.head(5)
# Feature 2: ratio of amount spending and avergae spending by Description of transaction
df2['ratio_coverage_by_state'] = df2['Average Covered Charges'] / df2.median_coverage_by_DRG_by_state
df2.head(2)
Out[21]:
In [22]:
# select DRG Definition = '039 - EXTRACRANIAL PROCEDURES W/O CC/MCC', city = 'New York'
selected1 = df2.loc[df2['Provider City'] == 'NEW YORK']
selected2 = selected1.loc[selected1['DRG Definition'] == '039 - EXTRACRANIAL PROCEDURES W/O CC/MCC']
There are 5 providers in New York provides '039 - EXTRACRANIAL PROCEDURES W/O CC/MCC' discharge, from the graph below we can see NYU Hospitals Centers have 1.8 times higher coverage on this clinical condition compared to the average level in New York State.
In [23]:
fig = px.scatter(selected2, x="Provider Name", y="ratio_coverage_by_state", color = 'ratio_coverage_by_state', size = 'ratio_coverage_by_state',
title = "Ratio of coverage for selected clinical condition to median in New York",
size_max=60,width=800, height=500)
fig.show()
Feature 3. ratio of average total payment to a provider/ median for this clinical condition in the state¶
- group by 'DRG Definition', 'Provider State', median('Average Covered Charges')
Why the feature can identify fraud:
- we can examine the actual payment to each provider in selected city for selected clinical condition, and identify outliers by the ratio.
In [24]:
# create variable: average_coverage_by_DRG_by_state
median_payment_by_DRG_by_state = df2.groupby(['DRG Definition', 'Provider State'])['Average Total Payments'].median()
# create average spending group by agency and merchant category
median_payment_by_DRG_by_state = pd.DataFrame(median_payment_by_DRG_by_state)
median_payment_by_DRG_by_state.columns = ['median_payment_by_DRG_by_state']
# add the new feature average spending to the dataset
df2 = pd.merge(df2, median_payment_by_DRG_by_state, how='left', on=['DRG Definition', 'Provider State'])
# Feature 2: ratio of amount spending and avergae spending by Description of transaction
df2['ratio_payment_by_state'] = df2['Average Total Payments'] / df2.median_payment_by_DRG_by_state
df2.head(2)
Out[24]:
In [25]:
# select DRG Definition = '039 - EXTRACRANIAL PROCEDURES W/O CC/MCC', city = 'New York'
selected1 = df2.loc[df2['Provider City'] == 'NEW YORK']
selected2 = selected1.loc[selected1['DRG Definition'] == '039 - EXTRACRANIAL PROCEDURES W/O CC/MCC']
There are 5 providers in New York provides '039 - EXTRACRANIAL PROCEDURES W/O CC/MCC' discharge.
From the graph below we can see New York Presbyterian hospital received approximately 1.5 times higher payment on the selected clinical condition compared to the average level in New York State.
In [26]:
fig = px.scatter(selected2, x="Provider Name", y="ratio_payment_by_state", color = 'ratio_payment_by_state', size = 'ratio_payment_by_state',
title = "Ratio of actual payment to hospital for selected clinical condition to median in New York",
size_max=60,width=800, height=500)
fig.show()
Feature 4. ratio of average total medicard coverage paid to a provider/ median medicare coverage for this clinical condition in the state¶
- group by 'Provider State', median('Average Covered Charges')
Why the feature can identify fraud: we can examine which provider received most medicare payment, and examine the possibility of medicare fraud.
In [27]:
# create variable: average_medicare_by_DRG_by_state
median_medicare_by_DRG_by_state = df2.groupby(['DRG Definition', 'Provider State'])['Average Medicare Payments'].median()
# create average spending group by agency and merchant category
median_medicare_by_DRG_by_state = pd.DataFrame(median_medicare_by_DRG_by_state)
median_medicare_by_DRG_by_state.columns = ['median_medicare_by_DRG_by_state']
# add the new feature average spending to the dataset
df2 = pd.merge(df2, median_medicare_by_DRG_by_state, how='left', on=['DRG Definition', 'Provider State'])
# Feature 2: ratio of amount spending and avergae spending by Description of transaction
df2['ratio_medicare_by_state'] = df2['Average Medicare Payments'] / df2.median_medicare_by_DRG_by_state
df2.head(2)
Out[27]:
In [28]:
# select DRG Definition = '039 - EXTRACRANIAL PROCEDURES W/O CC/MCC', city = 'New York'
selected1 = df2.loc[df2['Provider City'] == 'NEW YORK']
selected2 = selected1.loc[selected1['DRG Definition'] == '039 - EXTRACRANIAL PROCEDURES W/O CC/MCC']
There are 5 providers in New York provides '039 - EXTRACRANIAL PROCEDURES W/O CC/MCC' discharge.
From the graph below we can see BETH ISREAL MEDICAL CENTER received approximately 1.5 times higher payment on the selected clinical condition compared to the average level in New York State, and it is the highest compared to other hospitals.
In [29]:
fig = px.scatter(selected2, x="Provider Name", y="ratio_medicare_by_state", color = 'ratio_medicare_by_state', size = 'ratio_medicare_by_state',
title = "Ratio of medicare payment to hospital for selected clinical condition to median in New York",
size_max=60,width=800, height=500)
fig.show()
Feature 5. ratio of total number of discharge by provider to the avergae level of discharges in the same city¶
- group by 'Provider Id', 'Provider City' , sum('Total Discharges')
Why the feature can identify fraud:
- we can examine by selected city, which hospital has extreme number of total discharges. We can further examine the outliers.
In [30]:
# create variable: average_medicare_by_DRG_by_state
median_discharge_by_provider_by_city = df2.groupby(['Provider Id', 'Provider City'])['Total Discharges'].median()
# create average spending group by agency and merchant category
median_discharge_by_provider_by_city = pd.DataFrame(median_discharge_by_provider_by_city)
median_discharge_by_provider_by_city.columns = ['median_discharge_by_provider_by_city']
# add the new feature average spending to the dataset
df2 = pd.merge(df2, median_discharge_by_provider_by_city, how='left', on=['Provider Id', 'Provider City'])
# Feature 5
df2['ratio_average_discharge_by_provider_by_city'] = df2['Total Discharges'] / df2.median_discharge_by_provider_by_city
df2.head(2)
Out[30]:
In [31]:
# select city = 'New York'
selected1 = df2.loc[df2['Provider City'] == 'NEW YORK']
selected1.head(2)
Out[31]:
From the graph we can see, NYU Hospital Center has 10 times total discharges compared to the average level of discharges of NYC
In [32]:
fig = px.scatter(selected1, x="Provider Name", y="ratio_average_discharge_by_provider_by_city", color = 'ratio_average_discharge_by_provider_by_city', size = 'ratio_average_discharge_by_provider_by_city',
title = "Ratio of total number of discharge by provider to the median discharges in New York",
size_max=60,width=900, height=600)
fig.show()
3. Dimension reduction¶
- Standardize columns
- Apply PCA, choose number of components
3.1 Principal Component Analysis (PCA)¶
- apply dimension reduction to get lower dimensions
In [33]:
# Standardization
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.cluster import KMeans
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
df3 = df2.loc[:, ['ratio_coverage_by_state', 'ratio_payment_by_state',
'ratio_medicare_by_state', 'ratio_average_discharge_by_provider_by_city']]
# Standardize the data to have a mean of 0 and a variance of 1
X_std = StandardScaler().fit_transform(df3)
The graph below shows that
the first 3 components explain 96% of the variance in our data.
the first 2 components explain 76% of the variance in our data.
the first component explains 51% of variance in the data.
In [34]:
# Create a PCA instance: pca
pca = PCA(n_components=4)
principalComponents = pca.fit_transform(X_std)
# Plot the explained variances
features = range(pca.n_components_)
plt.bar(features, pca.explained_variance_ratio_)
plt.xlabel('PCA features')
plt.ylabel('variance %')
plt.xticks(features)
# Save components to a DataFrame
PCA_components = pd.DataFrame(principalComponents)
In [35]:
#plt.figure(gigsize = (10,8))
plt.plot(pca.explained_variance_ratio_.cumsum(), marker = 'o', linestyle = '--')
plt.xlabel('number of components')
plt.ylabel('cummulative explained variance')
Out[35]:
In [36]:
# plot first 2 compnents in 2 dimensional space
plt.scatter(PCA_components[0], PCA_components[1], alpha=.1)
plt.xlabel('PCA 1')
plt.ylabel('PCA 2')
Out[36]:
In [37]:
# view PCA compnenent
PCA_components
Out[37]:
4. Clustering by K-means¶
we will use k-means clustering to view the top 3 PCA components, given the first 3 components explain 96% of the variance in our data.
From the graph, we can see the elbow point is when k = 2 and k = 4, however we will also examine other number of clusters
examine optimal number of clusters¶
In [38]:
# determine the best number of clusters
ks = range(1, 10)
inertias = []
for k in ks:
# Create a KMeans instance with k clusters: model
model = KMeans(n_clusters=k)
# Fit model to samples
model.fit(PCA_components.iloc[:,:3])
# Append the inertia to the list of inertias
inertias.append(model.inertia_)
plt.plot(ks, inertias, '-o', color='black')
plt.xlabel('number of clusters, k')
plt.ylabel('inertia')
plt.xticks(ks)
plt.show()
write a function to get summary statistics table for each value of k¶
In [46]:
# write a function to get summary statistics for each k
def identify_outlier_by_kmeans(k, PCA_components, num_component):
from sklearn.cluster import KMeans
# implement k means
kmean = KMeans(n_clusters = k, random_state=1)
# fit data: we only use the first 3 components
kmean.fit(PCA_components.iloc[:,:num_component])
df_kmean = df2.copy()
# create new column and add it to original dataset
df_kmean['cluster']=kmean.labels_
# count by cluster
table1 = pd.DataFrame(df_kmean['cluster'].value_counts())
table1.columns = ['count']
# summary statistics
table2 = df_kmean.loc[:,['Total Discharges', 'Average Covered Charges', 'Average Total Payments', 'Average Medicare Payments',
'ratio_coverage_by_state','ratio_payment_by_state','ratio_medicare_by_state','ratio_average_discharge_by_provider_by_city',
'cluster']].groupby(['cluster']).mean()
table3 = pd.concat([table1, table2], axis = 1)
# return distribution by cluster and summary statistics
return table3
n_clusters = 4¶
- cluster 3 has the least number of observations, it has the highest dischargest
- we cannot identify outliers when k = 4
In [47]:
identify_outlier_by_kmeans(4, PCA_components, num_component=3)
Out[47]:
n_clusters = 5¶
- cluster 4 has the least number of observations, it also has the highest dischargest
- we cannot identify outliers when k = 5
In [48]:
identify_outlier_by_kmeans(5, PCA_components, num_component=3)
Out[48]:
n_clusters = 6¶
- cluster 1 has the least number of observations, it also has the highest dischargest
- we cannot identify outliers when k = 6
In [49]:
identify_outlier_by_kmeans(6, PCA_components, num_component=3)
Out[49]:
n_clusters = 7¶
- cluster 7 has the least number of observations, it also has the highest dischargest
- cluster 3 has the 2nd least number of observations (5%), and highest average ratio of total coverage, actual payment, and medicare payment, health providers in this group requires more attention.
In [50]:
output7 = identify_outlier_by_kmeans(7, PCA_components, num_component=3)
In [51]:
# group by cluster
# y: last 4 columns
output7['cluster'] = [1,2,3,4,5,6,7]
output7
Out[51]:
In [52]:
# multiple line plot
plt.plot( 'cluster', 'ratio_coverage_by_state', data=output7, marker='', markerfacecolor='blue', markersize=12, color='skyblue', linewidth=4, label="ratio_coverage_by_state")
plt.plot( 'cluster', 'ratio_payment_by_state', data=output7, marker='', color='orange', linewidth=2,label="ratio_payment_by_state")
plt.plot( 'cluster', 'ratio_medicare_by_state', data=output7, marker='', color='olive', linewidth=2, linestyle='dashed', label="ratio_medicare_by_state")
plt.plot( 'cluster', 'ratio_average_discharge_by_provider_by_city', data=output7, marker='', color='red', linewidth=2, linestyle='dashed', label="ratio_average_discharge_by_provider_by_city")
plt.legend()
plt.xlabel('cluster')
plt.title('Plot ratios by cluster')
Out[52]:
In [54]:
# # if we do not plot the variable about discharges
plt.plot( 'cluster', 'ratio_coverage_by_state', data=output7, marker='', markerfacecolor='blue', markersize=12, color='skyblue', linewidth=4, label="ratio_coverage_by_state")
plt.plot( 'cluster', 'ratio_payment_by_state', data=output7, marker='', color='orange', linewidth=2,label="ratio_payment_by_state")
plt.plot( 'cluster', 'ratio_medicare_by_state', data=output7, marker='', color='olive', linewidth=2, linestyle='dashed', label="ratio_medicare_by_state")
plt.legend()
plt.xlabel('cluster')
plt.title('Plot ratios by cluster')
Out[54]:
we found ratio_payment_by_state and ratio_medicare_by_state are very similar, so we only keep one of them, ratio_payment_by_state
In [55]:
plt.plot( 'cluster', 'ratio_coverage_by_state', data=output7, marker='', markerfacecolor='blue', markersize=12, color='skyblue', linewidth=4, label="ratio_coverage_by_state")
plt.plot( 'cluster', 'ratio_payment_by_state', data=output7, marker='', color='orange', linewidth=2,label="ratio_payment_by_state")
plt.legend()
plt.xlabel('cluster')
plt.title('Plot ratios by cluster')
Out[55]:
In [ ]:
# percentage of cluster 2
8310/163065
n_clusters = 8¶
- cluster 4 has the least number of observations (0.4%) and the highest discharges
- cluster 2 has the 2nd least number of observations (3.4%), average ratio of total coverage and actual payment, health providers in this group requires more attention.
In [56]:
output8 = identify_outlier_by_kmeans(8, PCA_components, num_component=3)
output8['cluster'] = [1,2,3,4,5,6,7,8]
output8
Out[56]:
In [ ]:
# percentage of cluster 2
# 5701/163065
In [57]:
# group by cluster
# y: last 4 columns
# multiple line plot
plt.plot( 'cluster', 'ratio_coverage_by_state', data=output8, marker='', markerfacecolor='blue', markersize=12, color='skyblue', linewidth=4, label="ratio_coverage_by_state")
plt.plot( 'cluster', 'ratio_payment_by_state', data=output8, marker='', color='orange', linewidth=2,label="ratio_payment_by_state")
plt.plot( 'cluster', 'ratio_medicare_by_state', data=output8, marker='', color='olive', linewidth=2, linestyle='dashed', label="ratio_medicare_by_state")
plt.plot( 'cluster', 'ratio_average_discharge_by_provider_by_city', data=output8, marker='', color='red', linewidth=2, linestyle='dashed', label="ratio_average_discharge_by_provider_by_city")
plt.xlabel('cluster')
plt.title('Plot ratios by cluster')
plt.legend(loc='upper left')
plt.show()
we found ratio_payment_by_state and ratio_medicare_by_state are very similar, so we only keep one of them, ratio_payment_by_state
In [58]:
# if we do not plot the variable about discharges
plt.plot( 'cluster', 'ratio_coverage_by_state', data=output8, marker='', markerfacecolor='blue', markersize=12, color='skyblue', linewidth=4, label="ratio_coverage_by_state")
plt.plot( 'cluster', 'ratio_payment_by_state', data=output8, marker='', color='orange', linewidth=2,label="ratio_payment_by_state")
plt.xlabel('cluster')
plt.title('Plot ratios by cluster')
plt.legend(loc='upper left')
plt.show()
n_clusters = 9¶
- cluster 8 (0.01%) and cluster 4 (9.8%) has the least number of observations and the highest discharges
- cluster 2 has the 3nd least number of observations (3.4%), and relatively highest average ratio of total coverage and actual payment, health providers in this group requires more attention.
In [59]:
output9 = identify_outlier_by_kmeans(9, PCA_components, num_component=3)
output9['cluster'] = [1,2,3,4,5,6,7,8,9]
output9
Out[59]:
In [60]:
# percentage of cluster 2
5620/163065
Out[60]:
In [61]:
# group by cluster
# y: last 4 columns
output9['cluster'] = [1,2,3,4,5,6,7,8, 9]
# multiple line plot
plt.plot( 'cluster', 'ratio_coverage_by_state', data=output9, marker='', markerfacecolor='blue', markersize=12, color='skyblue', linewidth=4, label="ratio_coverage_by_state")
plt.plot( 'cluster', 'ratio_payment_by_state', data=output9, marker='', color='orange', linewidth=2,label="ratio_payment_by_state")
plt.plot( 'cluster', 'ratio_medicare_by_state', data=output9, marker='', color='olive', linewidth=2, linestyle='dashed', label="ratio_medicare_by_state")
plt.plot( 'cluster', 'ratio_average_discharge_by_provider_by_city', data=output9, marker='', color='red', linewidth=2, linestyle='dashed', label="ratio_average_discharge_by_provider_by_city")
plt.xlabel('cluster')
plt.title('Plot ratios by cluster')
plt.legend(loc='upper left')
plt.show()
In [62]:
# if we do not plot the variable about discharges
# multiple line plot
plt.plot( 'cluster', 'ratio_coverage_by_state', data=output9, marker='', markerfacecolor='blue', markersize=12, color='skyblue', linewidth=4, label="ratio_coverage_by_state")
plt.plot( 'cluster', 'ratio_payment_by_state', data=output9, marker='', color='orange', linewidth=2,label="ratio_payment_by_state")
plt.plot( 'cluster', 'ratio_medicare_by_state', data=output9, marker='', color='olive', linewidth=2, linestyle='dashed', label="ratio_medicare_by_state")
plt.xlabel('cluster')
plt.title('Plot ratios by cluster')
plt.legend(loc='upper left')
plt.show()
In [63]:
# we found ratio_payment_by_state and ratio_medicare_by_state are very similar, so we only keep one of them, ratio_payment_by_state
plt.plot( 'cluster', 'ratio_coverage_by_state', data=output9, marker='', markerfacecolor='blue', markersize=12, color='skyblue', linewidth=4, label="ratio_coverage_by_state")
plt.plot( 'cluster', 'ratio_payment_by_state', data=output9, marker='', color='orange', linewidth=2,label="ratio_payment_by_state")
plt.xlabel('cluster')
plt.title('Plot ratios by cluster')
plt.legend(loc='upper left')
plt.show()
n_clusters = 10¶
- cluster 7 (0.01%) and cluster 10 (9.7%) has the least number of observations and the highest discharges
- cluster 5 has the 2nd least number of observations (6.8%), and relatively highest average ratio of total coverage and actual payment, health providers in this group requires more attention.
In [64]:
output10 = identify_outlier_by_kmeans(10, PCA_components, num_component=3)
output10['cluster'] = [1,2,3,4,5,6,7,8,9,10]
output10
Out[64]:
In [65]:
# percentage of clusters
7051/163065
Out[65]:
In [66]:
# group by cluster
# y: last 4 columns
# multiple line plot
plt.plot( 'cluster', 'ratio_coverage_by_state', data=output10, marker='', markerfacecolor='blue', markersize=12, color='skyblue', linewidth=4, label="ratio_coverage_by_state")
plt.plot( 'cluster', 'ratio_payment_by_state', data=output10, marker='', color='orange', linewidth=2,label="ratio_payment_by_state")
plt.plot( 'cluster', 'ratio_medicare_by_state', data=output10, marker='', color='olive', linewidth=2, linestyle='dashed', label="ratio_medicare_by_state")
plt.plot( 'cluster', 'ratio_average_discharge_by_provider_by_city', data=output10, marker='', color='red', linewidth=2, linestyle='dashed', label="ratio_average_discharge_by_provider_by_city")
plt.xlabel('cluster')
plt.title('Plot ratios by cluster')
plt.legend(loc='upper left')
plt.show()
In [67]:
# if we do not plot the variable about discharges
# multiple line plot
plt.plot( 'cluster', 'ratio_coverage_by_state', data=output10, marker='', markerfacecolor='blue', markersize=12, color='skyblue', linewidth=4, label="ratio_coverage_by_state")
plt.plot( 'cluster', 'ratio_payment_by_state', data=output10, marker='', color='orange', linewidth=2,label="ratio_payment_by_state")
plt.plot( 'cluster', 'ratio_medicare_by_state', data=output10, marker='', color='olive', linewidth=2, linestyle='dashed', label="ratio_medicare_by_state")
plt.xlabel('cluster')
plt.title('Plot ratios by cluster')
plt.legend(loc='upper left')
plt.show()
In [68]:
plt.plot( 'cluster', 'ratio_coverage_by_state', data=output10, marker='', markerfacecolor='blue', markersize=12, color='skyblue', linewidth=4, label="ratio_coverage_by_state")
plt.plot( 'cluster', 'ratio_payment_by_state', data=output10, marker='', color='orange', linewidth=2,label="ratio_payment_by_state")
plt.xlabel('cluster')
plt.title('Plot ratios by cluster')
plt.legend(loc='upper left')
plt.show()
5. Conclusion¶
the identified cluster when k = 10 has reasonable observations and differentiation compared with other clusters
cluster 5 contains 6.8% observations, in this cluster, health providers has 2 times higher actual total payment compared to the median level in this state, and 2.1 times higher coverage compared to state median